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Learning  from  Hints 


1.  INTRODUCTION 

It  is  evident  that  learning  from  examples  needs  all  the  help  it  can  get.  When  an 
unknown  function  /  is  represented  to  us  merely  by  a  set  of  examples,  we  are  faced 
with  a  dilemma.  We  would  like  to  use  a  model  that  is  sophisticated  enough  to  have 
a  chance  of  simulating  the  unknown  function,  yet  we  want  the  model  to  be  simple 
enough  that  a  limited  set  of  examples  will  suffice  to  ‘tune’  it  properly.  These  two 
goals  are  often  on  a  collision  course. 

One  established  method  of  tackling  this  problem  is  regularization  (Akaike, 
1969).  It  is  an  attempt  to  start  out  with  a  sophisticated  model  and  then  restrict 
it  during  the  learning  process  to  fit  the  limited  number  of  examples  we  have.  Thus 
we  have  a  simple  model  in  disguise,  and  we  make  it  as  simple  as  warranted  by  the 
resources.  The  hope  is  that  the  restriction  (simplification)  of  the  model  has  not 
rendered  /  impossible  to  simulate,  and  the  justification  is  that  this  is  the  best  we 
can  do  anyway  given  a  limited  set  of  examples. 

Another  method  for  tackling  the  problem  is  the  use  of  hints  (Abu-Mostafa, 
1990,1993)  as  a  learning  aid.  Hints  describe  the  situation  where,  in  addition  to  the 
set  of  examples  of  /,  we  have  prior  knowledge  of  certain  facts  about  /.  We  wish  to 
use  this  side  information  to  our  advantage.  However,  hints  come  in  different  shapes, 
and  the  main  difficulty  of  using  them  is  the  lack  of  a  systematic  way  of  incorporating 
heterogeneous  pieces  of  information  into  a  manageable  learning  process.  If  what  we 
know  about  /  is  that  it  is  scale-invariant,  monotonic  over  part  of  its  domain,  and 
represented  by  a  given  set  of  examples,  we  still  have  to  integrate  this  information 
before  we  can  learn  the  function.  This  paper  concerns  itself  with  the  development 
of  a  systematic  method  that  integrates  different  types  of  hints  in  the  same  learning 
process. 

The  distinction  between  regularization  and  the  use  of  hints  is  worth  noting. 
Regularization  restricts  the  model  in  a  way  that  is  not  based  on  known  facts  about  /. 
Therefore,  /  may  be  implementable  by  the  original  model,  but  not  by  the  restricted 
model.  In  the  case  of  hints,  if  the  unrestricted  model  was  able  to  implement  /,  so 
would  the  model  restricted  by  the  hints,  since  /  cannot  be  excluded  by  a  litmus 
test  that  it  is  known  to  satisfy.  We  can  apply  any  number  of  hints  to  restrict  the 
model  without  risking  the  exclusion  of  /.  The  use  of  hints  does  not  preclude,  nor 
does  it  require,  the  use  of  any  form  of  regularization. 
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